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Preface 


Software  process  improvement  is  not  usually  covered  in  standard  software  engineering 
textbooks.  However,  because  it  is  a  topic  of  great  interest  to  the  software  industry,  both 
faculty  and  students  should  be  familiar  with  it.  The  goal  of  this  package  is  to  provide 
the  basis  for  an  introductory  30  to  60  minute  lecture  on  the  software  process  and  its 
improvement. 

The  material  in  the  main  body  of  the  document  is  intended  primarily  for  instructors.  In 
conjunction  with  the  student-oriented  document  described  below,  it  provides  technical 
information  from  which  an  instructor  can  prepare  a  lecture.  Two  appendices  provide 
additional  information  for  instructors.  Appendix  A  is  a  brief  description  of  the  use  of  the 
software  process  material  in  a  university  software  engineering  class  at  the  University  of 
Texas  at  Austin.  This  classroom  example  might  be  used  as  an  illustration  for  students 
to  relate  the  process  concepts  to  their  own  software  development  work.  Appendix  B 
presents  the  Software  Engineering  Institute’s  (SEI)  process  maturity  questionnaire 
from  which  instructors  and  students  can  gain  some  insight  into  how  a  software  process 
is  assessed. 

A  document  titled  “Introduction  to  Software  Process  Improvement”  is  attached;  it  is 
intended  for  students.  It  describes  the  SEI  capability  maturity  model  (CMM),  the 
maturity  questionnaire,  and  SEI  procedures  that  are  based  on  the  questionnaire; 
software  process  assessment  and  software  capability  evaluation.  Instructors  may  photo¬ 
copy  this  document  and  distribute  it  to  students  to  augment  their  textbook.  (This  doc¬ 
ument  is  also  available  electronically  in  PostScript  format  via  the  Internet,  from  which 
students  can  print  their  own  copies.  For  details,  send  a  request  to  Internet  address 
education@sei  .cmu.edu. ) 

Finally,  the  package  contains  overhead  transparency  masters  that  instructors  may  find 
useful  in  delivery  of  their  lectures. 

How  to  Use  the  Materials 

These  materials  are  introductory  in  nature,  and  they  assume  that  the  student  audience 
is  a  beginning  software  engineering  class.  However,  these  materials  may  also  be  useful 
to  prepare  a  talk  for  computer  professionals  and  managers,  or  to  prepare  a  general  pro¬ 
fessionalism  talk  for  computer  science  and  engineering  students. 

Although  the  main  bodj,  of  this  package  is  intended  for  instructors,  it  may  be  distributed 
to  students  if  the  instructor  desires.  Industry  audiences,  for  example,  may  want  to  see 
the  additional  CMM  material  or  even  the  questionnaire  from  Appendix  B.  However, 


CMU/SEI-93-EM-8 
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many  of  the  maturity  questionnaire  concepts  will  not  be  familiar  to  college  students 
unless  they  have  had  work  experience  or  they  have  already  had  this  material  in  class,  so 
students  may  become  overloaded  by  the  terminology  contained  in  the  questionnaire  and 
other  supplementary  material. 

In  the  future  we  hope  to  add  advice  on  applying  the  process  improvement  concepts  to  a 
software  engineering  class  project,  as  well  as  additional  materials  on  software  process 
metrics. 

References 

The  student  document  does  not  contain  a  bibliography  or  citations  because  we  have 
assumed  that  undergraduate  students  are  unlikely  to  seek  additional  reading  unless 
specifically  assigned  by  the  instructor.  The  bibliography  in  the  instructor’s  document 
does  contain  all  the  references.  References  in  the  student  document  to  an  author’s  name 
can  usually  be  found  under  that  author’s  name  in  the  instructor’s  bibliography.  The 
surveys  of  software  process  maturity  levels  mentioned  on  page  4  may  be  found  in 
[Humphrey 89b].  References  for  the  process  improvements  at  Hughes  Aircraft, 
Raytheon,  and  NASA  are  in  [Humphrey91],  [Dion92],  and  [Humphrey92],  respectively. 
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Lecture  Notes  on 

Software  Process  Improvement 


1.  Overview  of  Software  Process  and  Quality  Improvement 

Today,  concern  for  quality  has  become  an  international  movement.  England  requires 
quality  programs  to  be  certified  and  audited.  Europe  will  soon  offer  certification  for 
software  development  companies  that  meet  the  standards  described  in  ISO  9000 
(International  Standards  Organization  number  9000)  [Arter92].  Japan  has  awarded  the 
Deming  Prize  for  years  [Masaaki86]. 

In  the  United  States,  the  Department  of  Commerce  and  NASA  give  major  awards,  such 
as  the  highly  coveted  Malcolm  Baldridge  Quality  Award  [Garvin91],  for  quality 
improvement.  Many  companies  have  begun  to  implement  quality  improvement  or  total 
quality  management  (TQM)  programs  throughout  the  company,  not  just  in  software 
development. 

1.1.  Fundamental  Process  and  Process  Management  Concepts 

Process  is  a  term  used  to  describe  the  people,  methods,  and  tools  used  to  produce  soft¬ 
ware  products.  Improving  the  quality  of  the  product  is  believed  to  be  based  on  improv¬ 
ing  the  process  used  to  develop  the  product.  Because  software  is  intangible  and  not 
subject  to  the  same  physical  constraints  as  hardware  and  many  manufacturing  prod¬ 
ucts,  defining  the  software  process  can  be  difficult. 

Software  engineering  process  is  defined  as  the  system  of  all  tasks  and  the  supporting 
tools,  standards,  methods,  and  practices  involved  in  the  production  and  evolution  of  a 
software  product  throughout  the  software  life  cycle.  Process-driven  software  develop¬ 
ment  implies  that  organizational  process  is  adapted  to  meet  project  and  product  quality 
goals.  Software  development  should  be  guided  by  an  explicit  process,  with  environment 
and  tools  integrated  to  support  this  process.  Process  definition  is  a  prerequisite  to  pro¬ 
cess  improvement.  Defined  processes  promote  collaboration  and  teamwork  by  making 
activities,  roles,  and  dependencies  visible.  Process  management  supports  improvement 
of  the  defined  process  through  measurement  and  feedback. 

Current  implementations  of  process  management  combine  three  steps:  definition,  con¬ 
trol,  and  improvement  [Card89].  Process  definition  provides  an  exact  description  for  the 
work  to  be  performed.  The  current  process  is  used  as  a  baseline  against  which  changes 
will  be  compared.  Process  control  works  to  keep  significant  quality  parameters  within 
some  predefined  limits.  Process  improvement  involves  analyzing  problems  for  root 
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causes  and  working  to  correct  them.  Product  quality  is  seen  as  a  result  of  continuously 
improving  process  quality.  A  process  is  said  to  be  under  statistical  control  if  its  future 
performance  can  be  predicted  within  established  statistical  limits  [Deming86]. 


Watts  Humphrey  incorporates  these  concepts  into  a  software-oriented  model  in  his  book 
Managing  the  Software  Process  [Humphrey87].  Based  on  work  begun  at  IBM,  this 
model  adapts  the  ideas  to  software  development,  providing  five  maturity  levels ,  shown 
in  Figure  1,  which  define  an  effective,  staged  progression  toward  a  statistically  con¬ 
trolled  software  process.  Humphrey’s  work  became  the  foundation  of  SEI  software  pro¬ 
cess  improvement  and  the  SEI  capability  maturity  model  (CMM)  [Paulk91,  Paulk93]. 


Figure  1.  Five  levels  of  software  process  maturity  [Weber91] 


The  basic  ideas  of  statistical  process  control  have  been  known  and  practiced  in  other 
areas  for  at  least  60  years.  However,  these  methods  represent  significant  change  for 


most  businesses  and  institutions. 
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tion  to  adapt  the  ideas  to  their  own  environment  gradually,  over  time.  This  is  not  a 
quick  fix  or  a  band-aid  approach  to  improving  software  quality. 

The  capability  maturity  model  defines  the  five  stages  or  levels  through  which  a  company 
must  move  in  order  to  improve  process  and  resulting  product  quality.  Each  level  of  the 
model  is  the  foundation  for  the  next,  so  it  is  important  not  to  try  to  move  too  quickly  or 
to  skip  levels.  Care  must  be  taken  that  measures  not  be  misused,  either  to  evaluate 
individuals  or  to  compare  dissimilar  projects.  Still,  the  benefits  experienced  by  organi¬ 
zations  that  have  applied  these  software  improvement  methods  have  outweighed  the 
costs.  The  productivity  and  quality  improvements  can  help  us  retain  our  lead  in  the 
global  software  marketplace. 

1.2.  Historical  Background 

Walter  Shewart,  a  physicist,  worked  at  AT&T  Bell  Labs  in  statistical  process  control  in 
the  1930s.  W.  Edwards  Deming  based  his  work  on  the  Shewart  improvement  cycle  (a 
sequence  of  four  steps  that  are  repeated  indefinitely;  see  Table  1),  which  he  successfully 
adapted  to  Japanese  industry  after  World  War  II.  Current  Japanese  management 
strategy  continues  to  focus  on  quality  improvement  because  the  Japanese  believe  that 
the  productivity  and  profit  improvements  will  follow  naturally.  Many  companies  are 
applying  the  ideas  of  quality  or  process  improvement  across  their  organization. 
Software  process  improvement  is  the  application  of  these  concepts  to  software  develop¬ 
ment. 


1.  Plan 

Define  the  problem 

State  improvement  objectives 

2.  Do 

Identify  possible  problem  causes 
Establish  baselines 
Test  changes 

3.  Check 

Collect  data 
Evaluate  data 

4.  Act 

Implement  system  change 
Determine  effectiveness 
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Table  1.  Shewart  improvement  cycle  [Deming86] 


Shewart’s  Plan-Act-Check-Do  paradigm  is  the  basis  for  the  SEI  process  improvement 
program.  Shewart’s  paradigm,  applied  to  both  the  software  product  and  process,  gener¬ 
ally  consists  of  the  following  activities: 

Plan  The  SEI  capability  maturity  model  is  a  general  framework  or  plan  for  devel¬ 
oping  five  increasingly  improved  levels  (initial,  repeatable,  defined, 
managed,  and  optimizing)  of  software  process  maturity.  Because  the  CMM  is 
designed  to  be  generic,  each  organization  must  customize  its  process 
improvement  plan  for  its  own  application(s),  environment,  and  company 
organization.  The  five  levels  are  designed  as  a  logical  progression,  so  each 
level  must  be  achieved,  in  order,  from  one  to  five.  It  is  not  possible  to  skip 
levels. 

Act  Because  software  is  not  produced  by  a  manufacturing  process,  software 

designers  must  both  strive  to  meet  the  users’  functional  requirements  for  the 
product  and  design  for  correct  impl  mentation  and  easy  maintainability. 

There  will  usually  be  many  examples  of  process  improvements  that  are 
needed.  Efforts  should  focus  cn  higL-leverage  points,  and  action  plans  to 
correct  the  defects  must  be  evaluated  for  effectiveness.  Software  tools  to 
automate  and  standardize  the  process  may  aid  in  institutionalizing 
improvements,  but  tools  are  not  a  cure-all. 

Check  Software  inspections  and  peer  reviews  are  the  major  product  control  mecha¬ 
nism  used.  Quantifiable  inspections  results  such  as  change  requests  provide 
the  foundation  for  measurable  process  control  and  improvement. 

Audits  are  the  most  usual  process  verification  process.  Auditors  need  to 
examine  not  only  whether  the  standards,  procedures,  and  tools  are  adequate, 
but  they  also  to  see  how  well  the  project  is  following  the  prescribed  process 
plans. 

Do  Software  quality  control  is  often  specified  both  by  the  customer  acceptance 

criteria  in  the  contract  or  requirements  specification  and  by  whether  the 
software  product  meets  written  standards.  Software  measures  are  used  to 
measure  product  quality  in  a  quantifiable  way.  The  SEI  has  already 
published  a  core  set  of  measures  IFlorac92)  which  can  be  used  as  a  basis  and 
enhanced  by  the  organization  as  needed,  though  these  measures  are  still 
under  active  development. 

The  most  common  process  quality  control  approach  is  tracking  actual  against 
expected  performance.  Causes  for  significant  deviation  from  the  plan  are 
found  and  corrected.  In  the  later  stages  of  the  maturity  model,  the  organiza¬ 
tion  strives  to  actively  prevent  problems  and  errors  rather  than  to  wait  to 
detect  them  in  the  later  phases  of  the  software  development  project. 
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2.  The  Software  Engineering  Institute  Capability  Maturity  Model 


The  basic  concept  of  a  maturity  framework  was  inspired  by  Crosby’s  quality  manage¬ 
ment  maturity  grid  and  its  five  evolutionary  stages  in  adopting  quality  practices 
[Crosby79].  This  maturity  framework  was  adapted  for  software  by  Radice  and  others  at 
IBM  [Radice85,  Radice88].  Humphrey  brought  the  maturity  framework  from  IBM  to 
the  SEI  in  1986,  adding  the  concept  of  maturity  levels.  Various  aspects  of  the  maturity 
model  are  described  in  SEI  technical  reports  [Fowler9Q,  Weber91,  Florac92]  and 
Humphrey’s  book  [Humphrey87]. 

Process  management  fundamentals  are  firmly  grounded  in  science  and  engineering 
principles.  They  have  been  strongly  influenced  by  the  work  on  statistical  process  control 
developed  by  leaders  such  as  Deming  and  Juran  [Deming86,  Juran88],  The  SEI  method 
incorporates  a  growing  body  of  experience  with  techniques  for  cost  and  size  estimation, 
configuration  management,  and  other  software  quality  improvement  approaches.  The 
field  of  technology  transfer  has  evolved  to  the  point  that  it  now  provides  operative 
methods  for  helping  organizations  adapt  to  technological  changes.  This  work  has  been 
combined  to  provide  a  foundaticn  for  learning  about  and  improving  the  software  devel¬ 
opment  process. 

The  software  field  is  young,  and  more  modem  tools  and  methods  will  evolve  over  time. 
Software  organizations  and  applications  are  far  too  diverse  to  fit  easily  into  a  single  pro¬ 
cess  model.  The  SEI  maturity  model  provides  a  framework  as  well  as  a  method  for 
evaluating  and  improving  the  software  engineering  process  within  an  organization.  The 
guidelines  do  not  mandate  particular  methodologies,  tools,  or  organizational  structure. 

2.1.  Uses  of  the  CMM 

The  capability  maturity  model,  developed  by  the  Software  Engineering  Institute,  is 
designed  to  help  both  development  organizations  and  customers  (government  organiza¬ 
tions  or  companies  who  acquire  software).  Software  organizations  need  to  understand 
the  quality  of  their  software  process  and  how  to  improve  it.  Organizations  contracting 
for  software  need  ways  to  evaluate  a  potential  contractor’s  capability  to  carry  out  the 
work. 

The  CMM  has  four  intended  uses  [Weber91]  to  help  organizations  improve  their  soft¬ 
ware  process  capabilities: 

1.  Identify  improvements 

2.  Identify  risks  in  selecting  contractors 

3.  Implement  a  process  improvement  program 

4.  Guide  definition  and  development  of  the  software  process 

Over  the  last  few  years,  the  SEI  has  expanded  and  refined  the  CMM  with  input  from 
many  professionals  from  both  government  and  industry.  The  current  version 
{Paulk93a,  Paulk93b]  is  based  on  several  years’  experience  applying  the  model  to  soflt- 
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(1)  (2)  (3) 


(4)  (5)  (6) 


(7) 


Figure  2.  Common  steps  in  SPAs  and  SCEs 

ware  process  improvement  {Humphrey 89b].  The  SEI  has  also  developed  two  specific 
methods  that  apply  the  CMM:  software  process  assessment  (SPA)  and  software  capabil¬ 
ity  evaluation  (SCE). 

A  software  process  assessment  is  an  in-house  determination,  primarily  of  the  weak¬ 
nesses  of  the  software  process  in  an  organization  as  a  whole.  It  is  an  internal  tool  that 
an  organization  can  choose  as  a  part  of  an  overall  program  for  improving  its  ability  to 
produce  high-quality  products  on  time  and  within  budget.  The  objectives  of  the  SPA 
method  are  to  (1)  identify  strengths,  weaknesses,  and  existing  improvement  activities 
on  which  to  base  an  organization-wide  improvement  effort  and  (2)  to  get  organizational 
buy-in  to  that  effort.  The  method  is  used  to  help  an  organization  identify  key  areas  for 
improvement,  begin  to  baseline  its  software  process,  and  initiate  improvements. 
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SCE 

Used  by  acquisition  organization 
for  source  selection  and  contract 
monitoring 

Results  to  the  organization  and 
the  acquirer 

Substantiates  current  practice 
Assesses  commitment  to  improve 


Analyzes  contract  performance 
potential 

Independent  evaluation — no 
organization  members  on  team 


Applies  to  performance  for  a 
particular  contract 


SPA 

Used  by  organization  to  improve 
software  process 

Results  to  organization  only 


Assesses  current  practice 

Acts  as  catalyst  for  process 
improvement 

Provides  input  to  improvement 
action  plan 

Collaborative — organization 
members  on  team,  with  represen¬ 
tative  from  licensed  SPA 
associate  or  SEI 

Applies  to  organization  overall, 
not  individual  projects 


Table  2.  Comparison  between  SPA  and  SCE 


A  software  capability  evaluation  is  an  independent  evaluation  of  an  organization’s  soft¬ 
ware  process  as  it  relates  to  a  particular  acquisition.  It  is  a  tool  that  helps  an  external 
group  (an  “acquirer”)  determine  the  organization’s  ability  to  produce  a  particular  prod¬ 
uct  having  high  quality  and  to  produce  it  on  time  and  within  budget.  The  objective  of 
the  SCE  method  is  to  identify  strengths,  weaknesses,  and  existing  improvement  activi¬ 
ties  in  a  supplier’s  software  process  that  best  indicate  the  risk  associated  with  using 
that  supplier  for  a  particular  software  acquisition.  The  method  is  used  to  identify  soft¬ 
ware  risks,  help  in  mitigating  these  risks,  and  motivate  initiation  of  improvement 
programs. 

The  common  steps  in  SPAs  and  SCEs  are  summarized  in  Figure  2.  Additional  similari¬ 
ties  and  differences  between  SPAs  and  SCEs  are  shown  in  Table  2  and  are  elaborated  in 
[SEI92], 

The  SEI  maturity  questionnaire  (see  Appendix  2)  is  used  for  both  SPA  and  SCE,  but 
there  are  differences  in  how  it  is  used  in  each  method.  These  differences  include  the 
objectives,  the  make-up  of  the  site  visitation  teams,  the  criteria  for  determining  scope 
and  for  defining  findings,  and  the  ownership  of  the  results.  In  view  of  these  differences, 
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the  outcomes  of  assessments  and  evaluations  are  not  likely  to  be  interchangeable  or 
directly  comparable.  The  SCE  team  may  conclude  that  a  particular  weakness  has  low 
risk  associated  with  it  for  the  acquisition  and  therefore  discount  it  (for  example,  subcon¬ 
tract  management  in  an  acquisition  that  may  not  involve  subcontracts),  while  a  SPA 
team  might  recommend  corrective  action  as  a  high  priority  for  the  same  weakness  (since 
other  projects  involve  subcontractors).  Both  views  would  be  valid  for  their  respective 
purposes. 

There  is  considerable  interest  in  the  software  community  in  using  assessments  and  in 
the  concept  of  continuous  quality  improvement  that  forms  the  basis  of  the  methodology. 
The  SEI  has  licensed  independent  training  and  consulting  companies  (SPA  associates) 
to  provide  training  and  assistance  in  applying  the  SPA  method  to  a  particular  environ¬ 
ment.  Software  process  conferences,  software  engineering  process  groups  in  companies, 
local  software  process  improvement  network  (SPIN)  groups,  and  tool  support  are  begin¬ 
ning  to  emerge. 

3.  Capability  Maturity  Model  Practices 

The  capability  maturity  model  describes  the  characteristics  of  a  mature  software  pro¬ 
cess.  The  model  also  shows  how  an  immature  software  process  can  evolve  into  a  well- 
managed  mature  one.  The  overall  structure  of  the  model  is  shown  in  Figure  3.  Major 
components  of  the  CMM,  as  shown  in  the  figure,  include: 

•  Maturity  level:  five  levels  or  plateaus  on  the  path  to  a  mature  software  process. 

•  Process  capability:  capability  refers  to  expected  results,  that  is,  what  can  we  pre¬ 
dict  from  this  organization’s  next  project  based  on  their  current  process  capability? 

•  Key  process  areas:  a  cluster  of  related  activities  that,  when  performed  collectively, 
achieve  a  set  of  goals  considered  important  for  enhancing  process  capability.  These 
each  contain  common  features. 

•  Goals:  the  high-level  objectives  to  be  achieved  by  the  key  practices  for  that  specific 
key  process  area. 

•  Key  practices:  the  policies,  procedures,  and  activities  that  most  significantly 
contribute  to  the  institutionalization  and  implementation  of  the  key  process  area. 

•  Questions:  yes/no  questions  that  sample  the  key  practices. 

Figure  4  shows  an  example,  or  a  particular  instantiation,  of  the  parts  of  the  CMM  struc¬ 
ture  to  illustrate  the  relationships  among  the  parts.  In  this  figure,  one  sees  the  relation¬ 
ship  between  the  components  (maturity  levels,  key  process  areas,  key  practices,  and 
questions).  At  the  repeatable  maturity  level,  in  the  key  process  area  of  software  project 
planning,  one  of  the  key  practices  is  to  estimate  project  size.  Thus,  a  typical  question 
from  the  maturity  questionnaire  might  be  “Do  you  use  a  documented  procedure  to  esti¬ 
mate  software  size?” 
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Maturity  Levels 


4.  Components  of  the  CMM 

Each  of  the  major  components  of  the  capability  maturity  model  is  described  in  more 
detail  below.  The  SEI  technical  report,  Key  Practices  of  the  Capability  Maturity  Model 
[Weber91],  elaborates  the  key  practices  that  correspond  to  each  maturity  level;  these 
practices  can  be  used  to  guide  both  software  process  improvements  and  capability 
evaluations. 
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Level  2,  Repeatable 


indicates  contains 
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Process 

Capability: 

disciplined 

process 


r  Key  Process  Area: 
Software  Project  Planning 


achieves  contains 


Goal: 

A  plan  is  developed  that 
appropriately  and 
realistically  covers  the 
software  activities  and, 
V  commitments 


specifies 

Key  Practice: 

- —  N 

Estimates  for  the  size  of 
software  products  are 
derived  according  to  a 
documented  procedure 


describes 


^  Activity:  Use  a 
documented  procedure  to 
estimate  software  size 
(e.g.,  lines  of  code,  > 
V  function  points) 


Figure  4.  Example  of  CMM  structure 

4.1.  Maturity  Level 

Each  maturity  level  in  the  CMM  indicates  a  certain  software  process  capability,  describ¬ 
ing  how  the  software  organization  is  expected  to  function:  initial  or  ad  hoc,  repeatable, 
defined,  managed,  or  optimizing.  Each  level  represents  an  improvement  in  the  software 
process.  An  organization’s  software  capability  can  be  improved  by  advancing  through 
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Level  2:  Repeatable 

Requirements  Management 

Software  Project  Planning 

Software  Project  Tracking  and  Oversight 

Software  Subcontract  Management,  if  applicable 

Software  Quality  Assurance 

Software  Configuration  Management 

Level  3:  Defined 

Organization  Process  Focus 
Organization  Process  Definition 
Training  Program 
Integrated  Software  Management 
Software  Product  Engineering 
Intergroup  Coordination 
Peer  Reviews 

Level  4:  Managed 

Process  Measurement  and  Analysis 
Quality  Management 

Level  5:  Optimizing 

Defect  Prevention 
Technology  Innovation 
Process  Change  Management 


Table  3.  Key  process  areas  (KPAs)  by  maturity  level  [Paulk91] 

these  five  stages  or  levels.  Each  maturity  level  allows  management  to  gain  a  better 
understanding  of  the  software  process.  Each  level  provides  the  foundation  necessary  to 
meet  the  goals  of  the  next  highest  maturity  level.  Each  maturity  level  has  been  decom¬ 
posed  into  parts  or  key  process  areas  as  shown  in  Figures  3  and  4. 

4.2.  Key  Process  Areas 

Key  process  areas  (KPAs)  identify  areas  on  which  an  organization  should  focus  in  order 
to  improve  its  software  development  processes.  Each  key  process  area  is  made  up  of  key 
practices  that  contribute  to  achieving  the  goals  of  the  KPA.  Goals  can  be  used  to  resolve 
whether  an  organization  or  project  has  adequately  implemented  a  key  process  area. 
Goals  signify  the  scope,  boundaries,  and  intent  of  each  key  process  area. 

Key  process  areas  are  building  blocks — fundamental  activities  for  organizations  trying 
to  improve  their  software  process.  Other  process  areas  exist,  but  these  were  selected  as 
particularly  effective  in  improving  process  capability.  Each  key  process  area  is  unique 
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to  a  single  maturity  level.  Table  3  shows  the  key  process  areas  required  for  each  matu¬ 
rity  level.  Note  that  there  are  no  KPAs  for  the  first  or  “initial”  level. 

4.3.  Key  Practices 

Key  practices  are  the  lowest  level,  specific  details  of  the  CMM.  Key  practices  define  each 
key  process  area  in  Table  3  by  specifying  policies,  procedures,  and  activities  that  con¬ 
tribute  to  satisfying  its  goal.  They  are  a  working  definition  of  the  key  process  area. 

Key  practices  provide  a  link  between  the  CMM  and  the  maturity  questionnaire.  Specific 
questions  relate  to  specific  key  practices.  Industry  experience  and  empirical  studies 
were  used  to  identify  the  key  practices  chosen  by  the  SEI.  Each  key  practice  describes, 
but  does  not  mandate,  how  that  practice  should  be  performed.  The  SEI  technical  report 
previously  cited  IWeber91]  provides  extensive  definitions  and  guidance  on  the  interpre¬ 
tation  of  key  practices. 

4.4.  Maturity  Questionnaire 

The  maturity  questionnaire  consists  of  questions  about  the  software  process  that  sample 
the  practices  in  each  key  process  area.  (See  the  example  question  in  Figure  4).  All  the 
questions  used  in  the  maturity  questionnaire  can  be  found  in  Appendix  B. 

The  maturity  questionnaire  is  a  springboard  for  an  assessment  or  evaluation  team’s 
visit.  The  CMM  provides  a  hierarchical  structure  that  guides  the  team  in  investigating 
an  organization’s  software  process.  Answers  to  the  questions  identify  process  strengths 
and  weaknesses  in  terms  of  key  process  areas.  Questions  in  the  maturity  questionnaire 
are  designed  to  determine  the  presence  or  absence  of  the  various  key  practices. 
Questions  are  not  open-ended,  but  are  intended  to  obtain  a  quantified  result  from  the 
following  answers:  yes,  no,  don’t  know,  and  not  applicable.  A  more  detailed  and  open- 
ended  questioning  process  begins  after  the  responses  to  the  maturity  questionnaire  have 
been  analyzed. 

The  SCE  team  identifies  strengths,  weaknesses,  and  improvement  activities  that  they 
consider  to  be  most  relevant  to  performance  on  the  acquisition  contract.  A  group  in  the 
acquisition  agency  then  transforms  the  findings  into  acquisition  risks  and/or  technical 
ratings,  which,  along  with  other  criteria,  the  agency  can  use  to  select  a  source  or  moni¬ 
tor  a  contract. 

The  SPA  team  also  analyzes  the  questionnaire  data  to  determine  the  current  software 
process  maturity  level,  identify  key  findings  (that  is,  determine  what  will  impede  capa¬ 
bility  to  produce  quality  software),  and  note  strengths  the  organization  can  build  upon. 
The  team  presents  the  results  to  senior  management  and,  often,  to  the  entire  organiza¬ 
tion  that  was  assessed.  The  team  often  enlists  the  aid  of  others  within  the  organization 
to  make  recommendations  for  process  improvement  actions.  An  action  planning  group 
(often  a  software  engineering  process  group,  under  the  guidance  of  a  management 
steering  committee)  develops  the  strategies  for  accomplishing  long-term  process 
improvement  and  determines  what  improvements  are  achievable  within  a  specific  time 
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frame.  They  work  with  many  others  in  the  organization  to  create  an  action  plan  and 
implement  it. 


5.  Conclusions 


While  great  strides  have  been  made  in  developing  software  engineering  methodologies 
and  techniques,  companies  have  been  unable  to  consistently  produce  high-quality  soft¬ 
ware.  Stories  of  software  problems  appear  on  a  regular  basis,  for  example 
[Neumann92].  Large  amounts  of  money  have  been  spent  on  projects  that  have  produced 
little  usable  software,  as  illustrated  graphically  in  the  results  of  a  General  Accounting 
Office  (GAO)  survey  shown  in  Figure  5. 


Software  that  could 
be  used  after  changes 
-3% 


Software  used  — 
but  later  reworked 
or  abandoned 
19% 


Software  that  could 
be  used  as  delivered 
-2% 


Software  paid  for 
but  not  delivered 
29.7% 


Software  delivered 
but  never  used 
47% 


Year  1982:  Nine  Contracts  Totalling  $6.8  Million 

Figure  5.  Results  of  GAO  survey  of  software  contracts 

Successful  projects  have  been  largely  based  on  individual  or  dedicated  team  effort 
rather  than  on  software  development  methods  [Humphrey89a].  We  have  come  to 
understand  that  benefits  of  better  methods  and  tools  cannot  be  realized  in  undisciplined 
projects.  In  the  typical  “firefighting”  mode  in  which  immature  organizations  function, 
software  quality  is  compromised  to  meet  unrealistic  schedules. 

Process  improvement  ideas  for  software  are  similar  to  current  business  practices  based 
on  total  quality  management,  popularized,  beginning  ten  years  ago,  by  books  such  as 
Quality  is  Free  [Crosby79]  and  In  Search  of  Excellence  fPeters82].  Many  of  the  process 
management  and  quality  improvement  concepts  have  been  adapted  from  the  work  sta¬ 
tistical  process  control  done  by  W.  Edwards  Deming  and  Joseph  Juran  lDeming86, 
Juran88,  Juran89].  The  SEI  developed  the  capability  maturity  model,  based  on  this 
earlier  work  by  quality  experts,  as  a  framework  for  evaluating  and  guiding  software 
process  improvement.  As  an  organization  increases  in  maturity,  the  difference  between 
targeted  and  actual  results  decreases  across  the  project.  Development  time  and  cost 
decrease,  while  productivity  and  quality  increase.  With  an  objective  basis  for  measuring 
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quality  and  setting  improvement  priorities,  time  and  costs  become  more  predictable  as 
rework  and  errors  are  removed  from  the  system  [Humphrey89a]. 

Companies  report  that  improvements  in  work  environment  and  motivation  have  turned 
out  to  be  an  even  greater  benefit  than  the  cost  saving  that  resulted  from  using  the  CMM 
[Henry92,  Mays90].  (See  also  Table  2  in  the  attached  student  document  Introduction  to 
Software  Process.)  In  a  mature  organization,  everyone  knows  the  processes  and  their 
own  responsibilities.  Workers  become  empowered  by  their  involvement  in  developing 
the  process  descriptions  and  by  the  ability  to  update  processes  as  needed.  Internal  pro¬ 
cesses  of  projects  become  more  visible.  Managers  know  current  project  status  and  can 
monitor  quality  and  customer  satisfaction. 

Software  process  issues  become  even  more  important  in  the  classroom,  where  the 
students  are  inexperienced.  Using  the  CMM  role  definitions  and  job  descriptions  in 
classes  can  ease  students’  transition  to  working  as  a  cohesive  software  engineering  team 
in  a  professional  environment.  They  have  a  better  appreciation  of  why  software  devel¬ 
opment  can  be  so  difficult,  and  they  have  a  meta-model  for  reducing  this  complexity. 
Even  without  applying  the  ideas  on  the  class  project,  students  can  gain  a  clearer  idea  of 
the  complications  inherent  in  developing  large  software  products  and  how  they  can  be 
managed.  Knowledge  of  software  process  concepts  has  proved  helpful  to  students  in 
talking  to  software  company  recruiters  as  well. 

Knowledge  transferred  from  the  university  to  industry  should  begin  to  include  process 
material  in  software  engineering  classes.  The  importance  of  software  process  is  vital 
information  needed  to  prepare  computer  science  students  for  the  challenges  of  modem 
software  technology. 
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Appendix  A:  Classroom  Experiences  with 

Software  Process 


Introduction 

Various  earlier  efforts  by  the  author  to  incorporate  software  engineering  techniques  into 
a  course  at  the  University  of  Texas  at  Austin  have  been  previously  described  [Werth88, 
Werth89,  Werth90,  Werth91].  Over  the  years,  using  higher  level  software  such  as 
MacApp,  HyperCard,  and  Oracle  in  a  Macintosh  II  laboratory,  we  developed  tools  for  the 
software  engineering  class  itself  to  provide  support  for  testing,  costing  version  control, 
analysis  and  design,  software  process  assessment,  and  others.  A  successful  collabora¬ 
tion  with  a  local  company  provided  valuable  experience  for  students  using  industry- 
strength  CASE  tools.  In  1992  wc  explored  the  area  of  software  process  improvement, 
both  by  teaching  the  foundations  and  by  applying  it  directly  to  the  class  project. 

Our  reasoning  was  that  if  software  process  improves  the  commercial  software  develop¬ 
ment  environment,  then  the  application  of  software  process  techniques  should  also 
strengthen  the  classroom  environment.  Two  positive  effects  could  be  expected.  First, 
successful  experience  with  the  techniques  on  the  class  project  would  result  in  students 
even  better  prepared  to  meet  the  challenges  of  modem  software  technology.  Second,  the 
use  of  quality  improvement  techniques,  applied  to  the  software  engineering  project  as 
currently  taught,  would  be  a  step  in  the  direction  of  improving  academic  education  as 
suggested,  for  example,  by  Peter  Denning  [Denning92]. 

Course  Description 

Starting  with  Tomayko’s  model  (Tcmayko87)  in  the  spring  1992  semester,  we  began  to 
develop  process  plans  and  tools  tor  the  class  project.  This  project  provided  an  early 
design  and  prototype  for  a  metrics  tool  to  collect  and  analyze  defect  reports,  as  described 
by  the  Software  Engineering  Institute  (SEI)  [Florac91].  In  the  fall  semester,  students 
employed  these  plans  and  tools  to  complete  the  design  and  implementation  of  Dante’s 
Defect  Tracker. 

The  spring  effort  was  based  on  disjoint  subteams  for  design,  implementation,  testing 
and  evaluation,  documentation,  configuration  management,  and  quality  assurance.  In 
the  fall,  we  integrated  the  process  teams  within  the  technical  teams,  providing  a  matrix 
management  scheme.  Each  of  the  four  technical  teams  (design,  implementation,  test¬ 
ing,  and  documentation)  elected  one  person  to  each  of  the  following  process  teams:  pro¬ 
ject  administration,  system  administration,  configuration  management,  quality  assur¬ 
ance,  and  documentation  specialists.  This  new  organization  seemed  to  match  more 
closely  the  roles  that  evolved  during  the  spring  semester’s  effort,  as  well  as  incorporate 
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natural  liaison  and  communication  of  process  procedures  within  the  technical  teams. 
Because  of  lack  of  experience,  undergraduates  find  it  easier  to  monitor  and  impose  con¬ 
trol  when  they  are  members  of  both  the  technical  and  the  process  teams. 

Technical  teams  met  and  developed  documents  appropriate  to  their  role:  functional  and 
design  specifications  by  the  design  team;  software  product  with  programmer’s  manual 
by  the  implementation  team;  test  plans,  test  cases,  results,  and  reports  by  test  and  eval¬ 
uation;  and  the  user’s  manual(s)  by  the  documentation  team.  Process  teams  developed 
or  enhanced  existing  plans  to  describe  their  process  function  and  wrote  process  legacies 
to  pass  along  their  increased  understanding  of  their  process  role(s)  for  future  teams. 

Each  Friday,  at  the  status  meeting  held  during  class,  one  technical  team  presented  their 
work,  while  process  teams  made  announcements  and  reported  progress.  Everyone  in 
the  class  acted  as  either  the  status  meeting  moderator  or  recorder  at  some  point  during 
the  course.  Agendas  and  minutes  were  sent  by  e-mail  to  class  members.  Status  meet¬ 
ings  worked  very  well,  greatly  improving  communication  and  student  process  learning, 
as  well  as  reducing  the  instructor  workload.  Our  industry  “user,”  Herb  Krasner, 
attended  these  meetings  as  his  schedule  permitted. 

Walkthroughs  or  reviews  were  held  during  the  week,  often  after  class  on  Monday  or 
Wednesday,  in  preparation  for  the  Friday  presentation.  Attendees  included  the  pre¬ 
senting  team  and  appropriate  representatives  from  related  technical  and  process  teams. 

The  configuration  control  board  (CCB)  consisted  of  each  team’s  configuration  manage¬ 
ment  representative,  along  with  the  head  of  quality  assurance,  the  external  auditor  (a 
teaching  assistant),  and  CEO  (the  instructor).  Overall  project  management  and  coordi¬ 
nation  gravitated  to  the  CCB  meetings,  held  in  the  lab  before  class.  Since  other  teams’ 
members  were  often  working  in  the  lab  during  this  time  and  all  teams  were  repre¬ 
sented,  many  questions  and  issues  were  resolved  quickly  and  easily.  Results  were 
announced  or  further  discussion  took  place  immediately  after  the  CCB  meeting,  during 
class. 

Lessons  Learned 

Student  learning  included  the  usual  lessons  such  as  the  discovery  of  the  complexity  of 
software  development,  the  need  for  communication  and  teamwork,  and  the  importance 
of  configuration  management.  Understanding  of  these  issues  was  considerably  deeper 
with  the  new  course  organization,  however. 

While  this  semester’s  class  was  relatively  low  in  work  experience  and  leadership  skill, 
and  relatively  high  in  weak  egos,  significant  learning  took  place  due  to  the  improved 
process  structure.  In  fact,  process  improvement  worked  well  enough  to  be  instrumental 
in  allowing  us  to  bypass  some  of  the  battles  between  certain  members  of  the  design  and 
implementation  teams.  More  centralized  project  management  may  help  here  also. 

Reducing  the  process  learning  startup  time  is  of  major  importance,  and  several 
improvements  are  under  development.  We  used  an  excellent  book,  The  Team  Handbook 
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[Scholtes89],  as  a  supplementary  process  text,  but  additional  class  time  needs  to  be  used 
to  practice  the  techniques.  Students’  skills  are  weak  enough  that  teamwork  cannot  be 
learned  solely  from  outside  homework  assignments.  The  students  themselves  suggested 
assignments,  quizzes,  or  other  means  to  ensure  that  everyone  learn  roles  and  processes 
early.  Informal  liaisons  between  the  technical  teams  may  need  to  be  made  explicit  and 
enforced.  Further  work  on  developing  written  job  descriptions  and  process  plans  will 
remedy  some  remaining  deficiencies. 

Technical  writing  and  presentation  skills  are  critical  to  process  improvement  and  need 
to  be  strengthened.  Some  science  college  and  departmental  efforts  in  this  direction  may 
help.  Our  department’s  Contemporary  Issues  in  Computer  Science  elective  would  be  an 
ideal  prerequisite,  as  it  is  designed  to  incorporate  writing  and  presentation  skills  within 
a  discussion  of  social,  ethical,  and  professional  issues.  Relegating  advanced  methodol¬ 
ogy  and  CASE  tool  training  to  additional,  probably  separate,  courses  will  allow  the 
course  to  concentrate  more  on  the  project  and  ensure  that  all  have  the  necessary  techni¬ 
cal  skills,  as  well  as  making  the  course  workload  more  equitable. 

These  techniques  lead  naturally  to  an  analysis  of  the  educational  environment  itself. 
When  processes  work  smoothly,  the  underlying  environment  and  infrastructure  weak¬ 
nesses  become  more  apparent.  Process  improvement  applied  to  the  students’  working 
environment  identifies  bottlenecks.  In  a  time  of  limited  resources,  this  is  especially 
helpful  for  guiding  instructors  in  directing  their  course  organization  efforts. 

Conclusions 

As  usual,  the  instructor  learned  more  than  the  students  during  the  semester.  Having 
explicit  process  roles  and  duties  greatly  improved  students’  learning  on  the  software 
engineering  project.  While  it  is  difficult  to  quantify,  individual  process  skills  and  expe¬ 
riences  seem  to  affect  the  quality  of  the  end-products  in  a  substantive  way.  The  techni¬ 
cal  analysis,  design,  and  testing  techniques  employed  are  important;  but  the  empower¬ 
ment,  shared  learning,  and  more  stable  environment  provided  by  quality  improvement 
efforts  seem  instrumental  in  increasing  the  learning  benefits  of  the  software  engineer¬ 
ing  project  course.  As  the  students  observed,  the  whole  is  indeed  greater  the  sum  of  the 
parts.  Learning  and  applying  software  process  can  improve  software  engineering  edu¬ 
cation  by  giving  students  a  meta-model  to  understand  and  help  them  manage  the  com¬ 
plexities  of  a  large  software  development  project.  Teaching  and  applying  software 
process  is  a  vital  part  of  increasing  the  amount  of  technology  transferred  as  students 
move  from  the  classroom  to  industry. 
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Appendix  B:  Software  Process  Assessment 

Questionnaire 


Five  levels  of  process  maturity  have  been  defined  for  the  assessment  of  software  engi¬ 
neering  organizations: 

Level  1  Initial 

Level  2  Repeatable 

Level  3  Defined 

Level  4  Managed 

Level  5  Optimized 

Level  1  -  Initial  Process  -  The  initial  environment  has  ill-defined  procedures  and 
controls.  While  positive  responses  to  some  of  the  organizational  questions  are  likely,  the 
organization  does  not  consistently  apply  software  engineering  management  to  the 
process,  nor  does  it  use  modern  tools  and  technology. 

Level  2  -  Repeatable  Process  -  At  Maturity  Level  2,  the  organization  uses  standard 
methods  and  practices  for  managing  software  development  activities  such  as  cost  esti¬ 
mating,  scheduling,  requirements  changes,  code  changes,  and  status  reviews.  The 
organization  will  provide  positive  responses  to  most  of  the  following  questions  (*  indi¬ 
cates  a  question  of  greater  importance  in  determining  the  CMM  level). 

1.1.1  For  each  project  involving  software  development,  is  there  a  designated 
software  manager? 

1.1.2  Does  the  project  software  manager  report  directly  to  the  project  (or  project 
development)  manager? 

*1.1.3  Does  the  Software  Quality  Assurance  (SQA)  function  have  a  management 
reporting  channel  separate  from  the  software  development  project 
management? 

*1,1.6  Is  there  a  software  configuration  control  function  for  each  project  that 
involves  software  development? 

1.2.2  Is  there  a  required  training  program  for  all  newly  appointed  development 
managers  designed  to  familiarize  them  with  software  project  management? 

1.3. 1  Is  a  mechanism  used  for  maintaining  awareness  of  the  state-of-the-art  in 

software  engineering  technology? 
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*2.1.3 

2.1.4 

2.1.5 

2.1.7 

2.1.9 
*2.1.14 
*2.1.15 
*2.1.16 

2.1.17 

2.2.1 

*2.2.2 

*2.2.4 

2.2.7 

2.2.8 

2.2.9 

2.2.10 
2.2.11 
2.2.12 
2.2.16 
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Is  a  formal  procedure  used  in  the  management  review  of  each  software 
development  prior  to  making  contractual  commitments? 

Is  a  formal  procedure  used  to  assure  periodic  management  review  of  the 
status  of  each  software  development  project? 

Is  there  a  mechanism  for  assuring  that  software  subcontractors,  if  any,  follow 
a  disciplined  software  development  process? 

For  each  project,  are  independent  audits  conducted  for  each  step  of  the 
software  development  process? 

Are  coding  standards  applied  to  each  software  development  project? 

Is  a  formal  procedure  used  to  make  estimates  of  software  size? 

Is  a  formal  procedure  used  to  produce  software  development  schedules? 

Are  formal  procedures  applied  to  estimating  software  development  cost? 

Is  a  mechanism  used  for  ensuring  that  the  software  design  teams  understand 
each  software  requirement? 

Are  software  staffing  profiles  maintained  of  actual  staffing  versus  planned 
staffing? 

Are  profiles  of  software  size  maintained  for  each  software  configuration  item, 
over  time? 

Are  statistics  on  software  code  and  test  errors  gathered? 

Are  profiles  maintained  of  actual  versus  planned  soft  ware  units  designed, 
over  time? 

Are  profiles  maintained  of  actual  versus  planned  software  units  completing 
unit  testing,  over  time? 

Are  profiles  maintained  of  actual  versus  planned  software  units  integrated, 
over  time? 

Are  target  computer  memory  utilization  estimates  and  actuals  tracked? 

Are  target  computer  throughput  utilization  estimates  and  actuals  tracked? 

Is  target  computer  I/O  channel  utilization  tracked? 

Are  software  trouble  reports  resulting  from  testing  tracked  to  closure? 
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2.2.18  Is  test  progress  tracked  by  deliverable  software  component  and  compared  to 
the  plan? 

2.2.19  Are  profiles  maintained  of  software  build/release  content  versus  time? 

*2.4.1  Does  senior  management  have  a  mechanism  for  the  regular  review  of  the 

status  of  software  development  projects? 

2.4.5  Is  a  mechanism  used  for  regular  technical  interchanges  with  the  customer? 

*2.4.7  Do  software  development  first-line  managers  sign  off  on  their  schedules  and 
cost  estimates? 

*2.4.9  Is  a  mechanism  used  for  controlling  changes  to  the  software  requirements? 

*2.4.17  Is  a  mechanism  used  for  controlling  changes  to  the  code?  (Who  can  make 
changes  and  under  which  circumstances?) 

2.4.20  Is  there  a  mechanism  for  assuring  that  regression  testing  is  routinely 
performed? 


Level  3  -  Defined  Process  -  At  Maturity  Level  3,  the  organization  not  only  defines  its 
process  in  terms  of  software  engineering  standards  and  methods,  it  also  has  made  a 
series  of  organizational  and  methodological  improvements.  These  specifically  include 
design  and  code  review,  training  programs  for  programmers  and  review  leaders,  and 
increased  organizational  focus  on  software  engineering.  A  major  improvement  in  this 
phase  is  the  establishment  and  staffing  of  a  software  engineering  process  group  that 
focuses  on  the  software  engineering  process  and  the  adequacy  with  which  it  is  imple¬ 
mented.  In  addition  to  the  questions  for  Level  2,  organizations  at  Level  3  will  respond 
“yes”  to  most  of  the  following  questions. 

1.1.4  Is  there  a  designated  individual  or  team  responsible  for  the  control  of 
software  interfaces? 

1.1.5  Is  software  system  engineering  represented  on  the  system  design  team? 

1.1.7  Is  there  a  software  engineering  process  group  function  ? 

1.2.1  Does  each  software  developer  have  a  private  computer-supported  work 
station/terminal? 

*1.2.3  Is  there  a  required  software  engineering  training  program  for  software 
developers? 

1.2.4  Is  there  a  required  software  engineering  training  program  for  first-line 
supervisors  of  software  development? 

*1.2.5  Is  a  formal  training  program  required  for  design  and  code  review  leaders ? 
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1.3.2 

*2.1.1 

2.1.2 

2.1.6 

2.1.8 

2.1.10 

2.1.11 

2.1.18 

*2.2.3 

*2.2.15 

*2.2.17 

2.4.3 

2.4.4 

*2.4.6 

2.4.8 

2.4.11 

*2.4.12 

*2.4.13 

2.4.14 


Is  a  mechanism  used  for  evaluating  technologies  used  by  the  organization 
versus  those  externally  available? 

Does  the  software  organization  use  a  standardized  and  documented  software 
development  process  on  each  project? 

Does  the  standard  software  development  process  documentation  describe  the 
use  of  tools  and  techniques? 

Are  standards  used  for  the  content  of  software  development  files/folders? 

Is  a  mechanism  used  for  assessing  existing  designs  and  code  for  reuse  in  new 
applications? 

Are  standards  applied  to  the  preparation  of  unit  test  cases? 

Are  code  maintainability  standards  applied? 

Are  man-machine  interface  standards  applied  to  each  appropriate  software 
development  project? 

Are  statistics  on  software  design  errors  gathered? 

Are  the  action  items  resulting  from  design  reviews  tracked  to  closure? 

Are  the  action  items  resulting  from  code  reviews  tracked  to  closure? 

Is  a  mechanism  used  for  identifying  and  resolving  system  engineering  issues 
that  affect  software? 

Is  a  mechanism  used  for  independently  calling  integration  and  test  issues  to 
the  attention  of  the  project  manager? 

Is  a  mechanism  used  for  ensuring  compliance  with  the  software  engineering 
standards ? 

Is  a  mechanism  used  for  ensuring  traceability  between  the  software 
requirements  and  top-level  design? 

Is  a  mechanism  used  for  ensuring  traceability  between  the  software  top-level 
and  detailed  designs? 

Are  internal  software  design  reviews  conducted? 

Is  a  mechanism  used  for  controlling  changes  to  the  software  design? 

Is  a  mechanism  used  for  ensuring  traceability  between  the  software  detailed 
design  and  the  code? 
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2.4.15 

*2.4.16 


2.4.18 


*2.4.19 

*2.4.21 

2.4.22 


Are  formal  records  maintained  of  unit  (module)  development  progress? 

Are  software  code  reviews  conducted? 

Is  a  mechanism  used  for  configuration  management  of  the  software  tools  used 
in  the  development  process? 

Is  a  mechanism  used  for  verifying  that  the  samples  examined  by  Software 
Quality  Assurance  are  truly  representative  of  the  work  performed? 

Is  there  a  mechanism  for  assuring  the  adequacy  of  regression  testing? 

Are  formal  test  case  reviews  conducted? 


Level  4  -  Managed  Process  -  At  Maturity  Level  4,  the  organization  typically  bases  its 
operating  decisions  on  quantitative  process  data,  and  conducts  extensive  analyses  of  the 
data  gathered  during  software  engineering  reviews  and  tests.  Tools  are  used  increas¬ 
ingly  to  control  and  manage  the  design  process  as  well  as  to  support  data  gathering  and 
analysis.  The  organization  is  learning  to  project  expected  errors  with  reasonable  accu¬ 
racy.  In  addition  to  questions  for  Levels  2  and  3,  organizations  at  Level  4  will  respond 
“yes”  to  most  of  the  following  questions. 


1.3.3 


2.1.12 
*2.1.13 
*2.2.5 
*2  2.6 
*2.2.13 
*2.2.14 
*2.3.1 

*2.3.2 

*2.3.3 


Is  a  mechanism  used  for  deciding  when  to  insert  new  technology  into  the 
development  process? 

Is  a  mechanism  used  for  managing  and  supporting  the  introduction  of  new 
technologies? 

Are  internal  design  review  standards  applied? 

Are  code  review  standards  applied? 

Are  design  errors  projected  and  compared  to  actuals? 

Are  code  and  test  errors  projected  and  compared  to  actuals? 

Are  design  and  code  review  coverages  measured  and  recorded? 

Is  test  coverage  measured  and  recorded  for  each  phase  of  functional  testing? 

Has  a  managed  and  controlled  process  database  been  established  for  process 
metrics  data  across  all  projects? 

Are  the  review  data  gathered  during  design  reviews  analyzed? 

Is  the  effort  data  from  code  reviews  and  tests  analyzed  to  determine  the  likely 
distribution  and  characteristics  of  the  errors  remaining  in  the  product? 
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*2.3.4  Are  analyses  of  errors  conducted  to  determine  their  process  related  causes? 
*2.3.8  Is  review  efficiency  analyzed  for  each  project? 

2.3.9  Is  software  productivity  analyzed  for  major  process  steps? 

*2.4.2  Is  a  mechanism  used  for  periodically  assessing  the  software  engineering 
process  and  implementing  indicated  improvements? 

2.4.10  Is  there  a  formal  management  process  for  determining  if  the  prototyping  of 
software  functions  is  an  appropriate  part  of  the  design  process? 

Level  5  -  Optimized  Process  -  At  Maturity  Level  5,  organizations  have  not  only 
achieved  a  high  degree  of  control  over  their  process,  they  have  a  major  focus  on  improv¬ 
ing  and  optimizing  its  operation.  This  includes  more  sophisticated  analyses  of  the  error 
and  cost  data  gathered  during  the  process  as  well  as  the  introducing  of  comprehensive 
error  cause  analysis  and  prevention  studies. 


*1.3.5  Is  a  mechanism  used  for  identifying  and  replacing  obsolete  technologies? 

*2.3.5  Is  a  mechanism  used  for  error  cause  analysis? 

*2.3.6  Are  the  error  causes  reviewed  to  determine  the  process  changes  required  to 
prevent  them? 

*2.3.7  Is  a  mechanism  used  for  initiating  error  prevention  actions? 

Technology  Addendum 

*3.1  Is  automated  configuration  control  used  to  control  and  track  change  activity 
throughout  the  software  development  process? 

3.2  Are  computer  tools  used  to  assist  in  tracing  software  requirements  to 
software  design? 

3.3  Are  formal  design  notations  such  as  PDL  used  in  program  design? 

3.4  Are  computer  tools  used  to  assist  in  tracing  the  software  design  to  the  code? 

*3.5  Is  the  majority  of  product  development  implemented  in  a  high-order 

language? 

3.6  Are  automated  test  input  data  generators  used  for  testing? 

3.7  Are  computer  tools  used  to  measure  test  coverage? 

3.8  Are  computer  tools  used  to  track  every  required  function  and  assure  that  it  is 
tested/verified? 
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3.9  Are  automated  tools  used  to  analyze  the  size  and  change  activity  in  software 
components? 

3.10  Are  automated  tools  used  to  analyze  software  complexity? 

3.11  Are  automated  tools  used  to  analyze  cross  references  between  modules? 

*3.12  Are  interactive  source-level  debuggers  used? 

*3.13  Are  the  software  development  and  maintenance  personnel  provided  with 
interactive  documentation  facilities? 

*3.14  Are  computer  tools  used  for  tracking  and  reporting  the  status  of  the  software 
in  the  software  development  library? 

3.15  Are  prototyping  methods  used  in  designing  the  critical  performance  elements 
of  the  software? 

3.16  Are  prototyping  methods  used  in  designing  the  critical  elements  of  the  man- 
machine  interface? 
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Attachments 


Two  attachments  follow.  The  first,  titled  “Introduction  to  Software  Process  Improve- 
ment,”  is  intended  as  supplementary  reading  for  students.  Its  pages  are  numbered 
separately  from  the  body  of  this  educational  materials  package. 

The  second  attachment  consists  of  nine  overhead  transparency  masters,  the  contents  of 
which  are  taken  from  the  body  of  this  document  and  from  the  student  document.  They 
include: 

•  Software  Process  Maturity  Levels  (Figure  1;  student  document  Figure  1) 

•  Shewart  Improvement  Cycle  (Table  1) 

•  Common  Steps  in  SPAs  and  SCEs  (Figure  2;  student  document  Figure  2) 

•  Comparison  Between  SCE  and  SPA  (Table  2;  student  document  Table  3) 

•  Capability  Maturity  Model  Structure  (Figure  3) 

•  Example  of  CMM  Structure  (Figure  4) 

•  Key  Process  Areas  by  Maturity  Level  (Table  4;  student  document  Table  1) 

•  Software  Systems  Development  is  Prone  to  Waste  (Figure  5) 

•  On-Board  Shuttle  Software  Improvement  (student  document  Table  2) 
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Introduction 

The  crisis  in  software  has  been  well  documented.  In  order  to  compete  successfully  in  the 
international  market,  we,  as  software  professionals,  need  to  improve  both  the  quality  of 
our  software  products  and  our  ability  to  work  within  time  and  budget  constraints. 
These  improvements  depend  strongly  on  process  as  well  as  technology. 

Modem  technology  can  help  us  combat  the  software  crisis,  yet  as  Fred  Brooks  warns  us, 
there  is  no  technological  “silver  bullet”  to  rescue  us.  Talented  people  are  important  in 
any  software  organization.  Nevertheless,  people  need  to  be  supported  by  a  good  working 
environment.  Software  development  is  hampered  by  changing  requirements,  unpre¬ 
dictable  schedules,  lack  of  standards,  and  insufficient  training  more  than  by  a  lack  of 
effort  on  the  part  of  professionals.  Curtis,  Krasner,  and  Iscoe  documented  these  issues 
effectively  in  their  report  describing  their  interviews  with  software  professionals.  In 
short,  problems  with  the  process,  rather  than  the  technology,  cause  a  substantial  num¬ 
ber  of  the  problems  in  software  development  and  maintenance. 

This  brief  document  begins  with  a  history  of  the  Software  Engineering  Institute  (SEI) 
and  its  software  process  work.  Terminology  is  introduced  and  the  five  levels  of  the  SEI 
capability  maturity  model  (CMM)  are  defined.  Possible  uses  and  future  directions  of  the 
model  are  provided. 

Background  and  Definitions 

The  Software  Engineering  Institute  was  established  at  Carnegie  Mellon  University  in 
Pittsburgh,  Pennsylvania  in  1984,  under  a  Department  of  Defense  contract.  Its  mission 
is  to  provide  leadership  in  advancing  the  state  of  the  practice  of  software  engineering  to 
improve  the  quality  of  systems  that  depend  on  software.  The  software  process  work 
began  two  years  later.  One  of  the  results  was  a  software  process  maturity  model.  In 
1987,  the  SEI  and  MITRE  Corporation  produced  the  first  maturity  questionnaire,  a  set 
of  yes-no  questions  that  address  organization  and  management  issues,  as  well  as  the 
technical  software  development  process.  Over  the  next  few  years,  the  SEI  developed 
two  methods  for  using  the  questionnaire  to  appraise  an  organization’s  software  process. 


This  document  is  taken  from  the  SEI  educational  materials  package  “Lecture  Notes  on  Software  Process 
Improvement”  by  Laurie  Honour  Werth,  document  number  CMU/SEI-93-EM-8,  copyright  1993  by  Carnegie 
Mellon  University.  Permission  is  granted  to  make  and  distribute  copies  for  noncommercial  purposes. 
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After  an  extensive  review  process,  the  capability  maturity  model  (CMM)  for  software 
replaced  the  software  process  maturity  model  in  1991.  The  CMM  summarizes  general 
software  process  practices  for  each  of  five  maturity  levels.  Once  the  current  level  of 
operation  is  established  using  the  maturity  questionnaire,  improving  a  company’s  soft¬ 
ware  process  involves  implementing  the  software  engineering  practices  needed  to  reach 
each  of  the  five  levels,  in  order,  from  lowest  to  highest. 

What  is  Software  Process  and  How  Can  It  be  Improved? 

It  is  important  to  understand  the  vocabulary  used  in  describing  the  software  process 
and  the  maturity  model.  These  terms  are  used  in  a  particular  way,  and  it  is  important 
to  know  their  meaning  in  order  to  comprehend  the  model.  It  is  also  necessary  to  under¬ 
stand  that  a  software  process  model  is  not  a  mathematical  formula.  In  this  context,  it  is 
a  description  of  how  to  conduct  the  process  of  software  development. 

Software  process  is  defined  as  a  set  of  activities  that  begin  with  the  identification  of  a 
need  and  concludes  with  the  retirement  of  a  product  that  satisfies  the  need;  or  more 
completely,  as  a  set  of  activities,  methods,  practices,  and  transformations  that  people 
use  to  develop  and  maintain  software  and  its  associated  products  (e.g.,  project  plans, 
design  documents,  code,  test  cases,  user  manuals). 

Software  process  capability  describes  the  range  of  expected  results  achieved  from  a 
software  process.  But  capability  is  not  the  same  as  performance.  Software  process  per¬ 
formance  is  the  actual  results  achieved  from  following  a  software  process.  That  is, 
results  achieved  (performance)  differ  from  results  expected  (capability). 

Many  software  process  techniques,  such  as  quality  assurance,  configuration  manage¬ 
ment,  inspections,  and  reviews,  are  described  in  most  software  engineering  textbooks 
and  will  not  be  covered  here.  From  a  general  problem-solving  point  of  view,  project 
management  may  be  described  simply  as: 

•  identifying  what  is  to  be  done 

•  deciding  how  to  do  it 

•  monitoring  what  is  being  done 

•  evaluating  the  outcome 

Most  managers  try  to  do  the  first  and  second  steps  above,  describing  the  what  and  how 
using  plans  and  schedules.  However,  even  though  managers  know  that  requirements 
will  change,  schedules  will  slip,  and  all  the  other  typical  problems  will  likely  occur  on 
the  project,  few  attempt  to  build  these  dynamic  events  into  their  plans.  In  addition, 
managers  need  to  monitor  project  activities  and  to  adjust  the  plans  as  modifications 
occur.  To  improve  the  software  development  process,  it  is  necessary  to  evaluate  the  suc¬ 
cess  of  the  project  and  avoid  repeating  problems  in  the  future.  The  CMM  addresses 
these  latter,  less  well-understood,  issues  in  an  effort  to  improve  the  software  develop¬ 
ment  process. 
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The  scientific,  closed-loop  model  of  management  can  be  explained  most  simply  as  fol¬ 
lows.  Project  plans  are  used  as  hypotheses,  and  project  results  are  evaluated  to  verify 
or  validate  these  hypotheses.  Statistical,  or  closed-loop,  process  management  is  based 
on  measurement.  First  a  baseline  is  determined.  After  improvements  are  instituted, 
measurements  are  repeated.  Results  are  compared  against  the  hypotheses  or  predic¬ 
tions  to  measure  progress.  This  process  comparison  is  repeated  with  the  goal  of  reduc¬ 
ing  the  differences  between  the  predicted  results  and  the  actual  results.  Thus,  the 
project  is  managed,  but  the  management  process  is  meta-managed. 

W.  E.  Deming,  one  of  the  pioneers  of  applying  statistical  process  control  in  industry, 
describes  process  improvement  as  a  continuous,  cycle  which  follows  these  steps: 

1.  Understand  the  status  of  the  development  process. 

2.  Develop  a  vision  of  the  desired  process. 

3.  List  improvement  actions  in  priority  order. 

4.  Generate  a  plan  to  accomplish  the  required  actions. 

5.  Commit  the  resources  to  execute  the  plan. 

6.  Start  over  at  step  1. 

SEI  Capability  Maturity  Model 

The  SEI  capability  maturity  model  is  derived  from  the  ideas  of  quality  improvement 
applied  to  software  development.  The  five-level  improvement  model  is  shown  in  Figure 
1.  The  five  stages  are  called  maturity  levels.  Each  represents  an  improvement  in  the 
software  process.  An  organization’s  software  capability  can  be  improved  by  advancing 
through  these  five  stages  or  levels.  The  CMM  helps  organizations  to  select  improve¬ 
ment  strategies  based  on  current  process  maturity  status  and  to  identify  critical  issues 
in  quality  and  process  improvement. 

The  following  descriptions  outline  the  primary  characteristics  of  the  software  process  for 
each  level  of  the  CMM. 

1.  Initial 

The  initial  software  process  is  characterized  as  ad  hoc.  Typically,  the  organization 
operates  without  formal  procedures,  cost  estimates  or  project  plans.  There  are  few 
mechanisms  to  ensure  that  procedures  are  followed.  Tools,  if  they  exist,  are  not  well 
integrated.  Change  control  is  lax  or  nonexistent.  Senior  management  neither  hears 
about  nor  understands  software  problems  and  issues.  Success  generally  depends  on 
the  efforts  of  individuals,  not  the  organization.  Not  too  surprisingly,  most  software 
organizations — 86%  in  an  early  survey  —  were  at  the  initial  level. 

2.  Repeatable 

At  the  repeatable  level,  project  controls  have  been  established  over  quality  assur¬ 
ance,  change  control,  cost,  and  schedule.  This  discipline  enables  earlier  successes  to 
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Figure  1.  The  five  levels  of  software  process  maturity 

be  repeated,  though  the  organization  may  have  problems  applying  these  techniques 
to  new  applications.  An  SEI  survey  found  16%  of  the  companies  at  the  repeatable 
level. 

3.  Defined 

The  defined  software  process,  for  both  management  and  engineering  activities,  is 
documented,  standardized,  and  integrated  across  the  organization.  The  process  is 
visible;  that  is,  it  can  be  examined  and  improvements  suggested.  Typicallj,  the 
organization  has  established  a  software  engineering  process  group  (SEPG)  to  lead  the 
improvement  effort,  keep  management  informed  on  progress,  and  facilitate  introduc¬ 
ing  other  software  engineering  methods. 

An  early  SEI  study  found  only  one  percent  of  organizations  surveyed  at  the  defined 
level,  though  more  recently  several  large  companies  have  achieved  this  stage  for 
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some  of  their  development  groups.  Organizations  need  to  achieve  the  defined  matu¬ 
rity  level  to  consistently  produce  quality  software  on  time  and  within  budget. 

4.  Managed 

Achieving  the  fourth,  or  managed,  level  requires  that  measures  of  software  process 
and  product  quality  be  collected  so  that  process  effectiveness  can  be  determined 
quantitatively.  A  process  database  and  adequate  resources  are  needed  to  continu¬ 
ally  plan,  implement,  and  track  process  improvements. 

5.  Optimizing 

At  the  optimizing  level,  quantitative  feedback  data  from  the  process  allows  continu¬ 
ous  process  improvement.  Data  gathering  has  been  partially  automated. 
Management  has  changed  its  emphasis  from  product  maintenance  to  process  analy¬ 
sis  and  improvement.  Defect  cause  analysis  and  defect  prevention  are  the  most 
important  activities  added  at  this  level. 

As  maturity  increases,  differences  between  targeted  and  actual  results  decrease;  costs 
decrease  and  development  time  shortens;  and  productivity  and  quality  increase.  The 
process  becomes  more  predictable  as  rework  is  prevented  and  the  risk  level  is  reduced. 
See  Table  1  for  the  specific  process  areas  added  at  each  maturity  level.  It  is  important 
to  note  that  maturity  levels  cannot  be  skipped  as  each  level  is  the  foundation  for  the 
next. 

Progress  through  the  maturity  levels  requires  high-level  management  backing  and  long¬ 
term  commitment.  It  requires  fundamental  modifications  in  the  way  managers  and 
software  practitioners  do  their  jobs,  and  this  takes  time  to  accomplish.  Software  process 
improvement  should  not  even  be  started  without  considerable  levels  of  corporate-wide 
encouragement  and  support. 

One  SEI  report  summarizes  the  capability  maturity  model  in  this  way. 

“The  CMM  provides  a  conceptual  structure  for  improving  the  manage¬ 
ment  and  development  of  software  products  in  a  disciplined  and  consis¬ 
tent  way.  It  does  not  guarantee  that  software  products  will  be  success¬ 
fully  built  or  that  all  problems  in  software  engineering  will  be  resolved. 

The  CMM  identifies  practices  for  a  mature  software  process,  but  it  is  not 
meant  to  be  either  exhaustive  or  dictatorial.  While  the  maturity  ques¬ 
tionnaire  samples  key  indicators  of  an  effect,  software  process  and  the 
CMM  identifies  the  characteristics  of  an  effective  software  process,  the 
mature  organization  addresses  all  issues  essential  to  a  successful 
project — including  people  and  technology — as  well  as  process.” 
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Repeatable 


Level  2: 

Requirements  Management 
Software  Project  Planning 
Software  Project  Tracking  and  Oversight 
Software  Subcontract  Management,  if  applicable 
Software  Quality  Assurance 
Software  Configuration  Management 

Level  3:  Defined 

Organization  Process  Focus 
Organization  Process  Definition 
Training  Program 
Integrated  Software  Management 
Software  Product  Engineering 
Intergroup  Coordination 
Peer  Reviews 

Level  4:  Managed 

Process  Measurement  and  Analysis 
Quality  Management 

Level  5:  Optimizing 

Defect  Prevention 
Technology  Innovation 
Process  Change  Management 


Table  1.  Key  process  areas  (KPA)  by  maturity  level 

How  Is  the  Maturity  Model  Used? 

Several  companies  have  already  reported  benefits  from  applying  the  capability  maturity 
model.  A  process  improvement  investment  of  $400,000  produced  an  annual  savings  of 
$2,000,000  at  Hughes  Aircraft.  Raytheon  saved  about  $9.2  million  by  eliminating 
rework  on  a  base  of  about  $115  million  in  software  development  costs.  At  IBM  Houston, 
the  NASA  space  shuttle  on-board  software  development  group  showed  the  results 
depicted  in  Table  2. 

There  are  two  major  ways  the  maturity  model  can  be  applied:  for  software  process 
assessment  (SPA)  and  for  software  capability  evaluation  (SCE).  Both  methods  are 
based  on  the  capability  maturity  model  and  maturity  questionnaire.  Together,  the 
model  and  questionnaire  provide  a  way  to  identify  and  compare  organizations’  strengths 
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NASA 


1982  1985 


Early  error  detection  (%  errors  found) 

48 

80 

Reconfiguration  time  (weeks) 

11 

5 

Reconfiguration  effort  (person-years) 

10.5 

4.5 

Product  error  rate  (errors  per  1000  lines  of  code) 

2.0 

0.11 

Table  2.  On-board  shuttle  software  improvements 

and  weaknesses.  Figure  2  shows,  at  a  high  level,  the  common  steps  in  the  SPA  and  SCE 
methods. 

There  are  differences  between  the  two  methods,  SPA  and  SCE,  as  summarized  in  Table 
3.  These  differences  include  the  objectives,  the  make-up  of  the  site  visitation  teams,  the 
criteria  for  determining  scope  and  for  defining  findings,  and  the  ownership  of  the 
results.  In  view  of  these  differences,  the  outcomes  of  assessments  and  evaluations  are 
not  likely  to  be  interchangeable  or  directly  comparable.  The  SCE  team  may  conclude 
that  a  particular  weakness  has  low  risk  associated  with  it  for  the  acquisition  and  there¬ 
fore  discount  it  (for  example,  subcontract  management  in  an  acquisition  that  may  not 
involve  subcontracts),  while  a  SPA  team  might  recommend  corrective  action  as  a  high 
priority  for  the  same  weakness  (since  other  projects  involve  subcontractors).  Both  views 
would  be  valid  for  their  respective  purposes. 

Software  Process  Assessment 

A  software  process  assessment  is  a  means  for  organizations  to  identify  their  strengths, 
weaknesses,  existing  improvement  activities,  and  key  areas  for  improvement.  It 
enables  them  to  determine  the  current  state  of  their  software  process  and  to  develop 
action  plans  for  improvement. 

Assessments  are  performed  by  a  team  of  6-10  experienced  software  professionals.  The 
majority  are  from  the  organization  being  assessed.  In  the  past,  individual  teams  have 
been  trained  by  the  SEI.  More  recently,  the  SEI  has  trained  and  licensed  SPA  associ¬ 
ates  to  provide  these  services  commercially.  A  SPA  associate  often  works  collaboratively 
with  team  members.  In  selected  cases,  the  SEI  does  so. 

An  organization  spends  two  to  six  months  (elapsed  time)  preparing  for  an  assessment, 
beginning  with  management  commitment  to  the  process  improvement  effort.  Other 
preparations  include  selecting  the  assessment  team,  and  selecting  the  representatives  of 
software  projects  and  functional  areas  (such  as  testing  and  quality  assurance)  who  will 
participate  in  the  on-site  assessment  activities. 
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Figure  2.  Common  steps  in  SPAs  and  SCEs 


The  assessment  team  helps  to  prepare  the  rest  of  the  organization  for  the  assessment 
and  for  the  ensuing  process  iirnrovement  activities.  In  addition  to  making  detailed 
plans  and  a  schedule,  they  tell  everyone  what  to  expect  and  concentrate  on  building 
support  for  process  improvement  and  the  assessment.  The  team  spends  five  days  on 
site. 

They  analyze  the  information  they  get  from  the  maturity  questionnaire,  individual 
interviews  with  project  leaders,  and  group  discussions  with  practitioners  (functional 
area  representatives).  To  encourage  interviewees  to  be  open,  the  SPA  team  treats  the 
information  as  confidential;  they  report  only  composite  findings  and  give  no  attribution 
to  individuals  or  projects.  Occasionally,  the  team  looks  at  documents  to  clarify  informa¬ 
tion  from  the  discussions. 
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SCE 

Used  by  acquisition  organization 
for  source  selection  and  contract 
monitoring 

Results  to  the  organization  and 
the  acquirer 

Substantiates  current  practice 
Assesses  commitment  to  improve 


Analyzes  contract  performance 
potential 

Independent  evaluation — no 
organization  members  on  team 


Applies  to  performance  for  a 
particular  contract 


SPA 

Used  by  organization  to  improve 
software  process 

Results  to  organization  only 


Assesses  current  practice 

Acts  as  catalyst  for  process 
improvement 

Provides  input  to  improvement 
action  plan 

Collaborative — organization 
members  on  team,  with  represen¬ 
tative  from  licensed  SPA 
associate  or  SEI 

Applies  to  organization  overall, 
not  individual  projects 


Table  3.  Comparison  between  SCE  and  SPA 


The  team  uses  the  maturity  model  to  help  them  categorize  data.  They  determine  the 
current  software  process  maturity,  identify  key  findings  (that  is,  they  determine  what 
will  impede  capability  to  produce  quality  software),  and  note  strengths  the  organization 
can  build  upon.  Decisions  are  made  by  consensus,  which  helps  to  counter  possible  bias 
in  any  individual’s  conclusions  about  the  data.  In  addition,  project  leaders  and  practi¬ 
tioners  who  were  interviewed  review  the  team’s  findings  and  give  feedback. 

The  team  presents  the  results  of  the  assessment  to  senior  management  and,  often,  to 
the  entire  organization  that  was  assessed.  They  recommend  what  can  be  done  to 
address  their  findings.  Many  teams  brainstorm  a  preliminary  list  of  recommendations 
and  enlist  others  in  the  organization  to  help  flesh  it  out.  Later,  the  team  writes  a  final 
report  that  presents  findings  and  recommendations  in  detail.  The  SEI  recommends  that 
the  report  be  widely  available  in  the  organization. 

After  the  assessment,  an  action  planning  group  (often  a  software  engineering  process 
group,  under  the  guidance  of  a  management  steering  committee)  develops  the  strategies 
for  accomplishing  long-term  process  improvement  and  determines  what  improvements 
are  achievable  within  a  specific  time  frame.  They  work  with  many  others  in  the  organi- 
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zation  to  create  an  action  plan  and  implement  it.  To  be  effective,  they  must  combine 
improvement  plans  with  other  organizational  plans  such  as  the  business  plan. 
Members  of  the  SPA  team  often  become  involved  in  these  activities;  their  involvement  is 
one  of  the  significant  differences  between  a  process  assessment  and  a  capability  evalua¬ 
tion. 


As  an  organization’s  software  process  matures,  periodic  reassessments  allow  them  to 
identify  new  priorities  and  strategies  for  further  improvement.  The  SEI  recommends 
reassessments  approximately  every  two  years. 

Software  Capability  Evaluation 

A  software  capability  evaluation  is  an  independent  evaluation  of  an  organization’s  soft¬ 
ware  process  as  it  relates  to  a  particular  acquisition.  It  is  a  tool  that  helps  an  external 
group  (an  “acquirer”)  determine  the  organization’s  ability  to  produce  a  particular  prod¬ 
uct  having  high  quality  and  to  produce  it  on  time  and  within  budget. 

Evaluations  are  performed  by  a  trained  and  experienced  team  of  4-6  members.  The 
members  are  not  part  of  the  organization  being  evaluated.  Rather,  they  come  from 
acquisition  organizations,  such  as  certain  government  agencies. 


An  SCE  team  looks  at  projects  that  perform  work  similar  to  that  required  by  the  new 
contract.  Thus,  the  selected  projects  are  usually  similar  to  the  acquisition  in  application 
domain,  size,  and  life-cycle  phases.  They  may  not  be  representative  of  the  organization 


as  a  whole. 


During  a  planning  period  of  several  weeks,  the  organization  being  evaluated  submits  a 
choice  of  projects  for  selection,  along  with  information  about  the  projects  and  about  the 
organization  in  general.  The  SCE  team  selects  projects,  reviews  high-level  documents 
from  the  organization,  and  makes  detailed  plans  and  a  schedule  for  the  site  visit. 


The  SCE  team  spends  three  days  on  site  interviewing  project  and  organization  person¬ 
nel,  one  at  a  time.  Team  members  may  talk  to  as  few  as  10  or  as  many  as  30  people. 
They  also  review  documents  from  one  to  four  projects,  depending  on  the  particular 
application  of  the  SCE  method  (contract  monitoring  or  source  selection).  As  in  the  SPA 
method,  the  SCE  team  does  not  attribute  information  to  individuals  but  holds  their 
responses  in  confidence. 


While  still  on  site,  the  team  identifies  strengths,  weaknesses,  and  improvement  activi¬ 
ties  that  they  consider  to  be  most  relevant  to  performance  on  the  acquisition  contract. 
Consensus  among  the  team  plays  a  major  role  in  countering  possible  bias  in  the  way  any 
individual  might  form  conclusions  about  the  data.  The  team  does  not  assign  a  maturity 
level,  but  rather  builds  a  profile  of  strengths  and  weaknesses  relevant  to  the  specific 
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The  SCE  team  presents  their  findings  to  the  organization  that  was  evaluated.  To  foster 
process  improvement,  the  SEI  strongly  recommends  that  the  presentation  include  ail 
the  detailed  SCE  findings  that  are  delivered  to  the  acquisition  agency. 

Ownership  of  the  SCE  findings  is  with  the  acquirer.  A  group  in  the  acquisition  agency 
transforms  the  findings  of  the  SCE  team  into  acquisition  risks  and/or  technical  ratings. 
To  ensure  objectivity  and  integrity,  the  SCE  team  deliberately  avoids  interpreting  its 
own  findings.  The  acquisition  agency  uses  the  SCE  findings  and  other  criteria  to  select 
a  source  or  monitor  a  contract. 

A  Final  Comment 

Software  process  maturity  requires  a  long-term  commitment  to  continuous  process 
improvement.  The  CMM  does  not  address  all  issues  important  for  successful  projects. 
The  CMM  does  not  imply  or  limit  the  choice  to  a  particular  life-cycle  model.  Specific 
software  technologies  such  as  reuse  or  prototyping  are  neither  required  nor  excluded. 
The  use  of  DoD-STD-2167A  is  not  dictated.  Projects’  documents  and  products  do  not 
have  to  match  the  CMM  exactly.  Particular  organizational  or  project  structures  are  not 
mandated.  Suggested  software  practices  are  generic,  intended  to  provide  flexibility. 
Each  project  or  organization  must  clarify  these  practices  for  its  specific  situation. 
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Software  Process  Maturity  Levels 


Ootimizina  (5) 


Initial  (1) 


Shewart  Improvement  Cycle 


1.  Plan 

Define  the  problem 

State  improvement  objectives 

2.  Do 

Identify  possible  problem  causes 
Establish  baselines 
Test  changes 

3.  Check 

Collect  data 
Evaluate  data 

4.  Act 

Implement  system  change 
Determine  effectiveness 


Common  Steps  in  SPAs  and  SCEs 


Capability  Maturity  Model  Structure 


7 - S - 

achieves  organized  by 


Example  of  CMM  Structure 


Maturity  Level: 


Level  2,  Repeatable 


indicates 


x — 

contains 


Process 
Capability: 
disciplined 
process 


Key  Process  Area: 
Software  Project  Planning} 


\ 

contains 


achieves 


Goat: 

A  plan  is  developed  that 
appropriately  and 
realistically  covers  the 
software  activities  andy 
commitments 


c 


\ 


,pL 

\ 


Key  Practice: 


Estimates  for  the  size  of 
software  products  are 
derived  according  to  a 
documented  procedure 


describes 


Activity:  Use  a 
documented  procedure  to' 
estimate  software  size 
(e.g.,  lines  of  code, 
function  points) 


Key  Process  Areas  by  Maturity  Level 


Level  2:  Repeatable 

Requirements  Management 

Software  Project  Planning 

Software  Project  Tracking  and  Oversight 

Software  Subcontract  Management,  if  applicable 

Software  Quality  Assurance 

Software  Configuration  Management 

Level  3:  Defined 

Organization  Process  Focus 
Organization  Process  Definition 
Training  Program 
Integrated  Software  Management 
Software  Product  Engineering 
Intergroup  Coordination 
Peer  Reviews 

Level  4:  Managed 

Process  Measurement  and  Analysis 
Quality  Management 

Level  5:  Optimizing 

Defect  Prevention 
Technology  innovation 
Process  Change  Management 


Software  Systems  Development  is  Prone  to  Waste 


~o 

0 


Year  1982:  Nine  Contracts  Totalling  $6.8  Million 


On-Board  Shuttle  Software 
Improvements 


NASA 

1982 

1985 

Early  error  detection 
(%  errors  found) 

48 

80 

Reconfiguration  time 
(weeks) 

11 

5 

Reconfiguration  effort 
(person-years) 

10.5 

4.5 

Product  error  rate 

(errors/1000  lines  of  code) 

2.0 

0." 

Comparison  Between  SCE  and  SPA 


SCE 

SPA 

Used  by  acquisition 
organization  for  source 
selection  and  contract 
monitoring 

Used  by  organization  to 
improve  software  process 

Results  to  the  organization 
and  the  acquirer 

Results  to  organization  only 

Substantiates  current 
practice 

Assesses  current  practice 

Assesses  commitment  to 
improve 

Acts  as  catalyst  for  process 
improvement 

Analyzes  contract 
performance  potential 

Provides  input  to 
improvement  action  plan 

Independent  evaluation — no 
organization  members  on 
team 

Collaborative — organization 
members  on  team,  with 
representative  of  licensed 
SPA  associate  or  SEI 

Applies  to  performance  for  a 
particular  contract 

Applies  to  organization 
overall,  not  individual 
projects 
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